System and method for indicating the quality of information to support decision making

ABSTRACT

A system for indicating the quality of a medical report including a medical report system which generates a medical score based on a clinician&#39;s interpretation of medical data, a portion of the medical report including text in a natural language; and a medical report grading device which processes the portion of text in natural language and generates one or more scores of one or more categories relating to the quality of the medical report.

The present application relates to clinical decision making. It findsparticular application in conjunction with clinical decision supportsystems and will be described with particular reference thereto.However, it is to be understood that it also finds application in otherusage scenarios and is not necessarily limited to the aforementionedapplication.

Clinical decision support systems (CDSS) have long been heralded aspivotal to improving the quality of health care. However, despitedecades of research and excellent results for the stand-aloneperformance (in terms of sensitivity and specificity) of such systems,successful introduction and application of such systems in real clinicalpractice are extremely limited. An explanation for the lack of successis the invalid assumptions about the decision making process thesesystems are supposed to support. Typically, most of the design andevaluation of clinical decision support systems is based on theprinciples of rational decision making. However, qualitative fieldresearch shows that the underlying assumptions of rational decisionmaking may bypass many aspects that impact the quality and efficiency ofdecision making in daily practice. One such an assumption of rationaldecision making is that the information and knowledge that goes in tothe decision making process (or system) is objective and can be directlyapplied without any preprocessing.

For example, successful communication through written radiology reportsdepends on how well the radiologist conveys the results of imageinterpretation and analysis. There are many reasons why a radiologist'sinterpretation may be sub-optimal and need to be reviewed. First, it isknown that the quality of readings of the radiologist are not uniformthroughout the day—with a bias towards better quality in the morningsthan in the evenings. Second, with many hospitals using tele-radiologyservices, the quality of radiology readings have more variations instyle that it affects interpretation. Third, the reports generated bytraining radiologists may not be optimal during their training periodsand may be in need for revision. In order to overcome this problem,typically, many radiology practices have a policy of re-reviewing aportion of their past scans. The problem with this approach is that thisstrategy does not ensure that all the scans that need a re-review areselected, thereby not ensuring complete quality control.

Currently, there are several indicators that may be used to indicate thequality and trustworthiness of reports or information used in theclinical decision support systems. For example, an indicator in medicalpractice often referred to by physicians is the seniority of a physicianthat produced the information, which may be rated by the number of yearsof practice. Other common examples to establish the quality andtrustworthiness of reports and information are rating systems ininternet applications (based on “wisdom of crowds” principle) that allowusers of particular information to rate how useful or reliable theinformation was to them. Another, widely accepted example from themedical practice to determine the credibility of information from aphysician is “case-volume”, i.e. the number times a physician hasproduced similar information in the past.

There are several shortcomings to the current solutions. The mostimportant shortcoming of the current practice is that establishingreliability involves a lot of human work and orchestration which causesinefficiency in the decision making process. Secondly, the indicatorsused, if any, are frequently extremely subjective (rating systems) oronly serve as weak proxy indicators (years of practice) withoutincorporating measurement on the actual produced information. Finally,establishing a more reliable proxy indicator such as case-volumerequires accurate registration. This is time consuming which leads toinefficiencies in healthcare. As such, volume measurements are onlyperformed for a few selected types of reports or procedures.

The present application provides new and improved methods and systemswhich overcome the above-referenced problems and others.

In accordance with one aspect, a system for indicating the quality of amedical report is provided. The system includes a medical report systemwhich generates a medical score based on a clinician's interpretation ofmedical data, a portion of the medical report including text in anatural language and a medical report grading device which processes theportion of text in natural language and generates one or more scores ofone or more categories relating to the quality of the medical report.

In accordance with another aspect, a method for indicating the qualityof a medical report is provided. The method includes receiving a medicalreport for a patient, a portion of the medical report being in a naturallanguage, processing the portion of the medical report in naturallanguage using a natural language processing engine, generating a scorethe medical report based on a plurality of categories relating to thequality of the medical report, and displaying the score of the medicalreport.

One advantage resides in the utilization of indicators to indicate thequality of reports and information.

Another advantage resides in the indication of the quality of thecontent, format, and trustworthiness of reports and information.

Another advantage resides in the reduction of human work involved inestablishing the content, format, and trustworthiness of reports andinformation.

Another advantage resides in establishing quality indicators without theneed for time consuming tailored registration procedures.

The invention may take form in various components and arrangements ofcomponents, and in various steps and arrangements of steps. The drawingsare only for purposes of illustrating the preferred embodiments and arenot to be construed as limiting the invention.

FIG. 1 is a block diagram of an IT infrastructure in accordance with thepresent application.

FIG. 2 is a visualization of a report card indicating the quality ofmedical report in accordance with the present application.

FIG. 3 is a flowchart diagram of a method for indicating the quality ofmedical reports in accordance with the present application.

With reference to FIG. 1, a block diagram illustrates one embodiment ofan information technology (IT) infrastructure 10 of a medicalinstitution, such as a hospital. The IT infrastructure 10 suitablyincludes one or more medical reporting systems 12, one or more patientmonitoring systems 14, one or more imaging systems 16, one or morelaboratory information systems 18, a patient information system 20, aclinical decision support system 22, and the like, interconnected via acommunications network 20. It is contemplated that the communicationsnetwork 24 includes one or more of the Intranet, a local area network, awide area network, a wireless network, a wired network, a cellularnetwork, a data bus, and the like.

The medical report system 12 generates medical reports for patients (notshown) cared for by the medical institution. The medical report system12 provides clinicians with the ability to interpret medical data andgenerate a medical report describing the physician's interpretation ofthe medical data. The medical report system 12 includes a display 26such as a CRT display and a liquid crystal display, to a computer, todisplay the medical data to be interpreted and a user input device 28such as a keyboard and a mouse, for the clinician to interpret themedical data and generate the medical report. The medical report system12 acquires the medical data from the one or more patient monitoringsystems 14, one or more imaging systems 16, one or more laboratoryinformation systems 18, a patient information system 20, and the like.The report is an electronic file in a format, such as PDF, DOCX, DOC,and so on. In some embodiments, newly generated patient data and/ornewly generated reports are saved in the IT infrastructure 10, such asin the patient information system 14. Further, in some embodiments,newly generated reports are electronically messaged to clinicians using,for example, email and/or printed using, for example, a laser printer,an inkjet printer and so on. The medical data suitable includesphysiological data, laboratory data, image data, and the like. Inresponse to the clinician requesting medical data, the medical reportsystem 12 acquires and displays the requested medical data. The medicalreport system 12 generates an electronic file of a medical report inresponse to receiving the interpretation from a clinician. The medicalreport system 12 records the interpretation in a natural language intothe electronic file, as well as links other medical data and informationinto the medical report. The medical report system 12 stores the medicalreport into a report database 30.

For example, the patient monitoring systems 14 obtain physiological datafor patients (not shown) cared for by the medical institution. Thephysiological data suitably includes data indicative of one or morephysiological parameters, such as heart rate, temperature, blood oxygensaturation, level of consciousness, concern, pain, urine output, and soon. Further, the patient data can be generated automatically and/ormanually. As to the former, sensors 32, such electrocardiographic (ECG)electrodes, blood pressure sensors, SpO2 sensors, and so on, measuringphysiological parameters of patients can be employed. As to the latter,user input devices 34 can be employed. In some embodiments, the patientmonitoring systems 14 include display devices 36 providing users a userinterface within which to manually enter the patient data and/or fordisplaying generated patient data to clinicians. The collectedphysiological data is concurrently transmitted to the patientinformation system 14 where the physiological data is displayed andstored. The medical report system 12 acquires and displays requestedphysiological data for a clinician to interpret. The medical reportsystem 12 generates an medical report, for example, a physiologicalstatus report, in response to receiving the interpretation of thephysician. Specifically, the medical report system 12 records thephysiological status report in a natural language into an electronicfile and stores the medical report into the report database 30.

Likewise, the imaging system 16 generates image data of a patientobtained using an image diagnosis apparatus 38 such as magneticresonance imaging systems, nuclear medical imaging system, CT scanners,ultrasound systems, fluoroscopy, and the like. The image diagnosisapparatus 38 captures an image of a patient. The imaging system 16generates image data reflecting the image of the patient and stores thegenerated image data to an image database 40. In some embodiments, theimaging system 16 include display devices 42 and a user interface 44 toadjust image reconstruction and acquisition parameters and/or fordisplaying generated image data to clinicians. The generated image datais also concurrently transmitted to the patient information system 14where the image data is displayed and stored. The medical report system12 acquires and displays requested image data for the clinician tointerpret. The medical report system 12 generates a medical report, forexample, a radiology report, in response to receiving the interpretationof the physician. Specifically, the medical report system 12 records theradiology report in a natural language into an electronic file andstores the medical report into the report database 30.

Similarly, the laboratory information system 18 generates laboratorydata from tests which are done on clinical specimens in order to getinformation relating to the health of a patient as pertaining to thediagnosis, treatment, and prevention of disease. The laboratory testingincluding hematological blood testing, coagulation laboratory testing,chemical blood, urine and body fluid testing, microbiology testing,urine laboratory testing, serological laboratory testing, cytology,histology, and pathology testing, immunohematology and blood bankingtesting, and the like. The laboratory information system 18 generateslaboratory data reflecting the laboratory tests and stores the generatedlaboratory data to a laboratory database 46. In some embodiments, thelaboratory information systems 18 include display devices 48 and a userinterface 50 within which to manually enter the laboratory data and/orfor displaying generated laboratory data to clinicians. The collectedphysiological data is concurrently transmitted to the patientinformation system 14 where the physiological data is displayed andstored. The medical report system 12 acquires and displays requestedlaboratory data for the clinician to interpret. The medical reportsystem 12 generates a medical report, for example, a pathology report,in response to receiving the interpretation of the clinician.Specifically, the medical report system 12 records the pathology reportin a natural language into an electronic file and stores the medicalreport into a report database 30.

It should also be appreciated that the medical report system 12 isutilized to generate a variety of medical reports suitable includinghistory and physical reports including a history of the present illness,past medical history, social history, and family medical history and anadmission diagnosis and a plan for the patient's treatment, aconsultation report including a brief history of the patient's illness,a specific physical exam depending on the particular type ofconsultation requested, and the consulting physician's impression andplan, an operative report including preoperative and postoperativediagnoses, the type of surgery or surgeries that were performed, thenames of the surgeon(s) and attending nursing staff, the type ofanesthesia and the name of the anesthesiologist, and a detaileddescription of the operative procedure itself, discharge summary reportincluding a radiologist's findings and impression, a pathology reportdescribing the findings of a tissue sample, laboratory report describesfindings of examinations of bodily fluids such as blood levels andurinalysis, other miscellaneous reports including hospital reports,cardiac catheterizations, electrophysiology studies,phacoemulsification, autopsies and psychological assessments, and thelike.

The patient information system 20 stores physiological data, image data,and laboratory data from the IT infrastructure 10, such as from the oneor more patient monitoring systems 14, one or more imaging systems 16,one or more laboratory information systems 18, in one or more databases52 of the IT infrastructure 10. The patient information system 20 alsostores medical reports generated by the medical report system 12 in theone or more databases 52 of the IT infrastructure. It is alsocontemplated that the patient information system 20 stores physiologicaldata, image, data, laboratory data, and medical reports generated fromother IT infrastructures. In some embodiments, the patient informationsystem 20 also stores physiological data, image data, laboratory data,and medical reports generated from user input devices 54 in the database52 and/or allows stored physiological data, image data, laboratory data,and medical reports to be viewed on display devices 56. Examples ofpatient information systems include, but are not limited to, electronicmedical record systems, departmental systems, and the like.

The CDSS 16 receives physiological data, image data, laboratory data,and/or medical reports from the IT infrastructure 10, such as from theone or more patient monitoring systems 14, one or more imaging systems16, one or more laboratory information systems 18, report medical system12 and/or the patient information system 14. It is also contemplatedthat the physiological data, image data, laboratory data, and/or medicalreports can be received from user input devices 58, optionally withdisplay devices 60 providing users a user interface within which toenter the physiological data, image data, laboratory data, and/ormedical reports. Using the physiological data, image data, laboratorydata, and/or medical reports, the CDSS 16 helps healthcare providersmake clinical decisions. Specifically, the CDSS 16 displays patienttreatment guidelines for a given patient in response to a query fromclinicians and physiological data, image data, laboratory data, and/ormedical reports. It is also contemplated that the CDSS perform or assistclinicians in clinical alerts and reminders, diagnostic assistance,prescription decision support, information retrieval, image recognitionand interpretation, therapy critiquing and planning, and the like.

The CDSS 16 also includes a grading device 62 which provides a rating ora score of the quality of the medical reports. It is also contemplatedthat the grading device 62 is located on the medical report system 12and/or the medical information server 14. Specifically, the gradingdevice utilizes natural language processing (NLP) engines to process andanalyze the medical report. After the medical reports are analyzed, thegrading device 62 scores the medical report on various categories andsub-categories all relating to the quality of the medical report. Thescoring of the medical reports are based on report content, reportformat, and report trustworthiness categories and sub-categories. Eachof the individual categories contributes a certain part to the metricsof an overall score or rating based on the quality, clarity, andaction-ability of the report, i.e. the three summary metrics are aweighted average of the individual scores of the categories andsub-categories. The grading device 62 ensures that sub-optimal medicalreports are detected and flagged. If the aggregated scores or individualscores are below a selected threshold, the report is selected forre-review The grading device 62 also provides a visualization of thescores for each medical report. The visualization assesses and indicatesthe quality of the report content, report format, and reporttrustworthiness and also the three merit factors including clarity,quality, and action-ability.

It is also contemplated that the grading device 62 automatically flagradiologist's readings that are determined to be less than optimal. Inanother embodiment, the grading device 62 can be used to a qualityassurance tool. For example, the grading device is utilized as a spelland grammar checker. For example, when a clinician finishes a medicalreport, the grading device 62 automatically checks the quality of thecontent of the report and alerts the clinician whether certain partsneed further elaboration.

Referring back to FIG. 1, the components of the IT infrastructure 10suitably include processors 64 executing computer executableinstructions embodying the foregoing functionality, where the computerexecutable instructions are stored on memories 66 associated with theprocessors 64. It is, however, contemplated that at least some of theforegoing functionality can be implemented in hardware without the useof processors. For example, analog circuitry can be employed. Further,the components of the IT infrastructure 10 include communication units68 providing the processors 64 an interface from which to communicateover the communications network 24. Even more, although the foregoingcomponents of the IT infrastructure 10 were discretely described, it isto be appreciated that the components can be combined.

With reference to FIG. 2, a report card visualization indicating thequality of a medical report is illustrated. The report card 100 includesa plurality of categories 102 and each of the categories includes aplurality of scored sub-categories 104. As illustrated, the report card100 grades a radiology medical report and include a report contentcategory 106, a report format category 108, and a report trustworthinesscategory 110. The report card 100 also includes a summary category 112which indicates the overall quality, clarity, and action-ability of themedical report. It is also contemplated that the report card 100includes more or less categories 102 and sub-categories 104 whichindicate the quality of medical reports. Additionally, although aradiology report card is illustrated it is also contemplated that therecord card indicates the quality of various types of medical reports.

As mentioned above, the grade of the medical report is based on reportcontent category 106, report format category 108, and reporttrustworthiness category 110. Each of the individual categoriescontributes a certain part to the metrics of quality, clarity andaction-ability of the report, i.e. the three summary metrics are aweighted average of the individual scores. The report content category102 includes a variety of sub-categories 104 that evaluate the qualityof the content of the medical report.

The sub-categories 104 of the report content include history andclinical information 114, comparison with previous studies 116,techniques of the scan 118, findings 120, recommendation for furthertest 122, and the like. The history and clinical informationsub-category 114 evaluates the content relevancy and the source of thecontent of the medical report. To evaluate the historical and clinicalinformation of the medical report, the grading device 62 utilizes a NLPto extract clinical information of the medical report and reasons forthe exam. This information is inputted into a matching algorithm thatestimates how well the clinical information section of a report matchesthe reasons for the exam. The grading device 62 provides a numericalscore indicating the quality of the content relevancy and the source ofthe content. The comparison with previous studies sub-category 116evaluates the content of the medical report with the content of similarpast medical reports. When comparing the content of the medical reportwith the content of previous studies, a NLP extracts the content of themedical report and a matching algorithm determines the similarity of thecontent between the current medical report and previous medical reportsthat are of a similar type. The grading device 62 provides a scoreindicated the similarity of the content of the current medical reportwith previous studies.

The technique of the scan sub-category 118 determines if the reportdescribes the type of exam, contrast agent, procedure-relatedinformation and details of any immediate or delayed proceduralcomplications and the management of the scan. To determine the techniqueof the scan, a NLP extracts DICOM headers (modality, protocol) from theimage data and the description of protocols in the Techniques(Procedures) section and matches them accordingly. Depending on theexistence of the information, the score for the technique of the scansub-category is given according to the quality of the description of thetechnique of the scan. The findings sub-category 120 describes preciseanatomical location, size, extent, shape of the findings. To describethe finding, existing medical ontologies containing comprehensivemedical terms, including RadLex, SNOMED-CT, BIRADS are utilized. Theontologies are integrated into a NLP to extract medical terms fromnarrative reports. When radiological findings are detected by NLP in apiece of text, the grading device automatically checks whether there areoccurrences of anatomical terms and measurements (size, shape andextent) in the surrounding text. If no anatomical terms and measurementsoccur the system assigns a score indicating the incompleteness of thefindings. The recommendation for further testing sub-category 122determines how the report will contribute to the diagnosis andmanagement of the patient's current clinical problem. To determine howmuch the medical report contributes to the current clinical problem, aNLP evaluate the severity of the findings. Findings like cancer,carcinoma, invasive, and the like are reported to referrals. If thosefindings are present in a medical report, the system can check whetherfollow up action are also present. The grading device 62 scores thecontribution of the medical report to the current clinical problem.

The sub-categories 104 of the report format include length 124, template126, communication adherence 128, confidence and certainty 130, and thelike. The length sub-category 124 determines how concise is the reportwhile still conveying the information required to highlight key findingsand to answer a clinical question. In order to accomplish this, a NLPconsiders the mean and median length of different radiology reportsduring a learning phase. During the learning phase, the systemclassifies the reports as optimal or as sub-optimal. The length of thecurrent report under evaluation are compared against the mean and medianlengths of other medical reports within the same category. A score isdisplayed based on how far from the mean is the current report toindicate the quality of length of the medical report. The templatesub-category 126 determines if the report follows a specific templatefor standardized examination and disease process. Medical reports ofcertain categories follow specific medical report templates. Todetermine if the medical reports uses the correct template, a NLPcompares the current medical report to the set of existing templates tosee how well it fits into the suggested templates using a matchingalgorithm. The grading device 62 provides a score to indicate whetherthe medical report utilizes a correct template.

The communication adherence sub-category 128 evaluates the how accuratethe medical report utilizes terminology with commonly agreed meaning anduse of anatomically specific lesion location. To determine communicationadherence, a NLP extracts the terms used in the medical report againstmedical ontologies to make sure that all the terms used adhere to thecommon terminology used for each disease process. If a certain term isdetected that is not part of the dictionary, then that term will beflagged. A score is then provided to indicate the quality of thecommunication adherence of the medical report. The confidence andcertainty sub-category 130 determines the confidence of the statementsin the medical report. Typically, longer and more complex sentences tendto convey impression of less certainty than shorter and simplyconstructed sentence. To determine confidence and certainty, a NLPsearches and finds the use-frequency of modifies like “might beconsistence”, “possibly represents”, “clinical information correlated”.A table of confidence/certainty words/phrases is collected from a corpusand computer linguistics tools like keyword matching and regularexpression to detect occurrences of the defined words/phrases themedical report. A score is then provided based on the occurrence ofdetected confidence/certainty words/phrases to indicate the confidence acertainty of the medical report.

The sub-categories 104 of the report trustworthiness include anexperience sub-category 132 and a proficiency sub-category 134. Theexperience sub-category automatically establishes how many times aproducer of a report (information) has produced the same type of report(information) before. A score is then provided to indicate theexperience of the producer of the medical report. The proficiencysub-category automatically establishes an indication for the quality ofinformation of that producer based on a comparison to past outcomesachieved based on similar information generated by the same producer.The grading device 62 provides a score based on the comparison toindicate the proficiency of the producer of the medical report.

For example, when determining the trustworthiness of a radiology orpathology report, a NLP analyzes a current radiology report matches thetext of the radiology report to known concepts in the ontology (forexample reporting templates from the RSNA or College of AmericanPathologists). This identifies (1) which type of cases the report isdealing with (e.g. the staging of a lung cancer tumor) and (2) theactual value of the radiology determined stage for the lung cancer.Based on the matched concept, all reports from the same user areanalyzed to identify how many times the clinician produced a similarreports (lung cancer staging report). This score is presented to thereader of the report in the report card 100 as a measure for theproficiency of the producer of the clinical report. Additionally, all ofthe medical records for all patients for which the radiologist produceda staging report in the past are analyzed in order to determine anindicator for quality for the radiologist's reported lung cancer stagein the past. This analysis uses a more elaborate ontology for the sameconcept, for example, clinical practice guidelines for lung cancer inwhich the radiology determined stage is related to the pathologydetermined stage which serves as gold standard benchmark. It is thendetermined how many times there is a mismatch between radiology stageand gold standard pathology stage. This score is presented to the readerof the radiology report as a measure for the proficiency of the producerof the stage information.

As another example, when determining the trustworthiness of achemotherapy treatment proposal. A NLP processes a current treatmentproposal created by a physician and matches the text to the concepts inthe ontology (for example a clinical practice guidelines). Thisidentifies which type of treatment proposal the plan is dealing with(e.g. a chemotherapy applied to reduce tumor size). Based on the matchedconcept, all reports from the same clinician are analyzed to identifyhow many times the clinician made a similar treatment decision. Thisscore is presented to the reader/reviewer of the treatment plan in thereport card. Additionally, the medical records for all patients forwhich the clinician earlier made similar decisions in the past areanalyzed in order to determine an indicator for quality for thephysician's treatment decision. This analysis uses a more elaborateontology for the same concept, for example, clinical practice guidelinesin which the treatment is related to anticipated tumor size reduction.It is then determined to which degree the tumor size reduction achievedover all patients from this particular clinician compares to theanticipated benchmark value for tumor size reduction embedded in theguideline. This score presented to the reader of the report card as ameasure for the proficiency of the producer of the decision.

In the summary category 112, the scores from the differentsub-categories 104 are weighted and averaged. The weighing of each ofthe different sub-categories may be predetermined based on specificclinical applications or adjusted by a reading clinician based on his orher preference. It should also be appreciated that the reading clinicianmay disable the automatic scoring 136 and manually input a sub-categoryscore if the clinician deems necessary. From the weighted and averagedscores, the grading device 62 determines scores which represent of theclarity of the report, quality of the report, and action-ability of thereport which are presented to the reader of the report card. The reportcard can also indicate a rating of the overall medical report. Forexample, the overall rating of the medical report may includevisualization 140 which indicates the overall quality of the medicalreport. It should also be appreciated that if any of the scores orrating of the medical report is below a selected threshold, the medicalreport is selected for re-review and reason for re-review is shown.

1. With reference to FIG. 3, a method 200 for indicating the quality ofmedical reports is provided. In a step 202, a medical report for apatient is received including a portion of the medical report being in anatural language. In a step 204, the portion of the medical report innatural language is processed using a natural language processingengine. In a step 206, a score of the medical report is generated basedon a plurality of categories relating to the quality of the medicalreport. In a step 208, the score of the medical report is displayed.

As used herein, a memory includes one or more of a non-transientcomputer readable medium; a magnetic disk or other magnetic storagemedium; an optical disk or other optical storage medium; a random accessmemory (RAM), read-only memory (ROM), or other electronic memory deviceor chip or set of operatively interconnected chips; an Internet/Intranetserver from which the stored instructions may be retrieved via theInternet/Intranet or a local area network; or so forth. Further, as usedherein, a processor includes one or more of a microprocessor, amicrocontroller, a graphic processing unit (GPU), anapplication-specific integrated circuit (ASIC), a field-programmablegate array (FPGA), and the like; a user input device includes one ormore of a mouse, a keyboard, a touch screen display, one or morebuttons, one or more switches, one or more toggles, and the like; and adisplay device includes one or more of a LCD display, an LED display, aplasma display, a projection display, a touch screen display, and thelike.

The invention has been described with reference to the preferredembodiments. Modifications and alterations may occur to others uponreading and understanding the preceding detailed description. It isintended that the invention be constructed as including all suchmodifications and alterations insofar as they come within the scope ofthe appended claims or the equivalents thereof.

Having thus described the preferred embodiments, the invention is nowclaimed to be:
 1. A system (10) for indicating the quality of a medicalreport, said system (10) comprising: a medical report system (12) whichgenerates a medical score based on a clinician's interpretation ofmedical data, a portion of the medical report including text in anatural language; and a medical report grading device or processor (62)which processes the portion of text in natural language and generatesone or more scores of one or more categories relating to the quality ofthe medical report.
 2. The system (10) according to claim 1, wherein theone or more categories include at least one of report content, reportformat, and report trustworthiness.
 3. The system (10) according toeither one of claims 1 and 2, wherein each of the categories includesone or more sub-categories which are scored by the medical reportgrading device or processor (62).
 4. The system (10) according to anyone of claims 1-3, wherein each of the scores of the one or morecategories and/or sub-categories are weighted and aggregated to generateone or more scores relating to at least one of the quality, clarity, andaction-ability of the medical report.
 5. The system (10) according toany one of claims 1-4, wherein an overall rating of the medical reportis calculated from at least one of the scores of the one or morecategories and/or sub-categories and the one or more scores relating toat least one of the quality, clarity, and action-ability.
 6. The system(10) according to any one of claims 1-5, wherein the medical reportgrading device (62) flags sub-optimal portions of the medical reportsfor review.
 7. The system (10) according to any one of claims 1-6,wherein the trustworthiness of the medical reports is scored based onthe experience and proficiency of the producer of the medical report. 8.The system (10) according to any one of claims 1-7, wherein the medicalreport grading device (62) generates and displays a visualizationincluding at least one of the scores of the one or more categoriesand/or sub-categories, the one or more scores relating to at least oneof the quality, clarity, and action-ability of the medical report, andthe overall rating of the medical report.
 9. A system for indicating thequality of a medical report, the system comprising: one or moreprocessors programmed to: process a portion of a medical report innatural language using a natural language processing engine; andgenerate a score of the medical report based on a plurality ofcategories relating to the quality of the medical report.
 10. The systemaccording to claim 9, wherein the one or more processors are furtherprogrammed to: weight and aggregate scores in the plurality ofcategories to generate an overall score indicative of quality, clarity,and action-ability of the medical report.
 11. A method for indicatingthe quality of a medical report, the method comprising: receiving amedical report for a patient, a portion of the medical report being in anatural language; processing a portion of the medical report in naturallanguage using a natural language processing engine; generating a scoreof the medical report based on a plurality of categories relating to thequality of the medical report; displaying the score of the medicalreport.
 12. The method according to claim 11, wherein the one or morecategories include at least one of report content, report format, andreport trustworthiness.
 13. The method according to either one of claims11 and 12, wherein each of the categories includes one or moresub-categories which are scored.
 14. The method according to any one ofclaims 11-13, wherein each of the scores of the one or more categoriesand/or sub-categories are weighted and aggregated and further including:to generating one or more scores relating to at least one of thequality, clarity, and action-ability of the medical report.
 15. Themethod according to any one of claims 11-14, further including:calculating an overall rating from at least one of the scores of the oneor more categories and/or sub-categories and the one or more scoresrelating to at least one of the quality, clarity, and action-ability.16. The method according to any one of claims 11-15, further including:flagging sub-optimal portions of the medical reports for review.
 17. Themethod according to any one of claims 11-16, wherein the trustworthinessof the medical reports is scored based on the experience and proficiencyof the producer of the medical report.
 18. The method according to anyone of claims 9-15, further including: generating and displaying avisualization including at least one of the scores of the one or morecategories and/or sub-categories, the one or more scores relating to atleast one of the quality, clarity, and action-ability of the medicalreport, and the overall rating of the medical report.
 19. One or moreprocessors (64) programmed to perform the method (200) according to anyone of claims 11-18.
 20. A non-transitory computer readable medium (66)carrying software which controls one or more processors (64) to performthe method (200) according to any one of claims 11-18.